Researching Data Science Education: Perspectives on Qualitative Research Methods

Dr. Allison Theobold

Today’s Layout


  1. Some background on qualitative research
  1. Qualitative investigations into student’s code
  1. Qualitative investigations into group work
  1. Implications for oral assessments

http://bit.ly/qual-research-duke

A bit about me…

Ph.D. in Statistics from Montana State University

“Supporting Data-Intensive Environmental Science Research: Data Science Skills for Scientific Practitioners of Statistics”

Land Acknowledgement

Duhram sits on the territory of several Native nations, including the Tutelo and Saponi speaking peoples. We acknowledge, respect, and thank the tribes on whose stolen land we are guests.

Indigenous people are not relics of the past. We who work and live here must acknowledge past violence and ongoing harm produced by the ongoing effects of colonization.

Qualitative Research

“Qualitative researchers strive to understand the meaning people have constructed about their world and their experiences.” (Merriam 2002)



“Qualitative research is an effort to understand situations in their uniqueness as part of a particular context and the interactions there. This understanding is an end in itself.” (Patton 1990)

What are the principles of qualitative research?


  • The researcher is the primary instrument for data collection and data analysis

  • The analysis seeks to find emerging themes

  • The product of a qualitative study is richly descriptive

How might this look?


Sample Selection

Select a sample from which the most can be learned!

Data Collection

Major sources of data – interviews, observations, documents

Data Analysis

Compares units of data to find common patterns across the data

Investigating Student Learning through Code

Warm-up (90 seconds)


RPMA2GrowthSub$Weight[RPMA2GrowthSub$Age == 1]


How would you describe the action(s) being taken in this statement?

A framework for analyzing student’s code (Schulte 2008)

Text Surface Program Execution Function
Macrostructure Understanding the overall structure of the program Understanding the “algorithm” of the program Understanding the goal / purpose of the program (in its context)
Relations References between blocks, e.g., method calls, object creation Sequence of method calls, object sequence diagrams Understanding how sub-goals are related to goals, how function is achieved by subfunctions
Blocks Regions of interest (ROI) that syntactically or semantically build a unit Operation of a block, a method, or a ROI (as a sequence of statements) Function of a block, may be seen as a sub-goal
Atoms Language elements Operation of a statement Function of a statement, only understandable in context

Coding student’s code


RPMA2GrowthSub$Weight[RPMA2GrowthSub$Age == 1]


Descriptive Code

“Filters a vector of values using extraction operator, based on an equality relation with a variable selected from dataframe using $ operator”

In-vivo Code

“Uses [ ] and == to filter vector, uses $ to select variable”

How could this be used?

Learning Trajectory

How does a student’s concept model of a dataset inform how they filter data?

Program Environment

How do the visualizations produced by students who learn ggplot differ from those who learn “base” R?

Linguistic Structure

How do the “evocative names” given to tidyverse functions impact learners’ mental models of what the function accomplishes?

Why is this important for data science education?


How can we distinguish merely interesting learning from effective learning (Wiggins and McTighe 2005)?

Investigating Power in the Classroom

Another Warm-up 🙃

Read through the dialogue on the back page of the handout. Cfonsider how students are using language to:

  • Build community and / or collaboration
  • Position (mathematical) thinking as significant

Discorse Analysis

Discourse Analysis is the study of language (structure, form, and syntax) together with the study of “language-in-use… [where we study] language in terms of actual utterances or sentences in speech or writing in specific contexts of speaking and hearing or writing and reading… Discourse allows us to create and enact our identities” (Gee 2014, 19–20).

Language is used to build:

  • Significance
  • Practices
  • Culturally supported activities
  • Identities
  • Relationships
  • Politics
  • Connections
  • Sign Systems and Knowledge

How could this look?

How do students use language and actions to (1) establish a collaborative environment, and (2) position each other’s thinking as significant while working in small groups on mathematical problem-solving tasks?

“The more one talks and the less one listens, the more likely it is that one’s viewpoint will function as if it were community consensus even if it is not” (Montell 2019).

“Fights over who gets to speak and whose words are recognized are indicative of power and status” (Johnson, 2002)


The Influence Framework (Engle, Langer-Osuna, and McKinney de Royston 2014)

  • the negotiated merit of each participant’s arguments

  • each participant’s intellectual authority

  • each participant’s access to the conversational floor

  • each participant’s degree of spatial privilege

A Discourse Scorecard

What can qualitative research teach us about oral exams?

(Theobold 2021)

Questions?

References

Engle, Randi A., Jennifer M. Langer-Osuna, and Maxine McKinney de Royston. 2014. “Toward a Model of Influence in Persuasive Discussions: Negotiating Quality, Authority, Privilege, and Access Within a Student-Led Argument.” Journal of the Learning Sciences 23 (2): 245–68. https://doi.org/10.1080/10508406.2014.883979.
Gee, James Paul. 2014. How to Do Discourse Analysis. 2nd ed. Abingdon, Oxon: Routledge.
Merriam, Sharan B. 2002. Qualitative Research in Practice: Examples for Discussion and Analysis. 1st ed. New York: John Wiley & Sons.
Montell, A. 2019. Wordslut: A Feminist Guide to Taking Back the English Language. New York: HarperCollins Publishers.
Patton, Mary Q. 1990. Qualitative Evaualuation Methods. 2nd ed. Thousand Oaks: Sage.
Schulte, Carsten. 2008. “Block Model.” Proceedings of the Fourth International Workshop on Computing Education Research, September. https://doi.org/10.1145/1404520.1404535.
Theobold, Allison S. 2021. “Oral Exams: A More Meaningful Assessment of Students’ Understanding.” Journal of Statistics and Data Science Education 29 (2): 156–59. https://doi.org/10.1080/26939169.2021.1914527.
Wiggins, G., and J. McTighe. 2005. Understanding by Design. 2nd ed. Alexandria: Association for Supervision; Curriculum Development (ASCD).